Statistical Approaches to the Model Comparison Task in Learning Analytics

Comparing the performance of predictive models of student success has become a central task in the field of learning analytics. In this paper, we argue that research seeking to compare two predictive models requires a sound statistical approach for drawing valid inferences about comparisons between the performance of such models. We present an overview of work from the statistics and machine learning communities and evaluate several methodological approaches, highlighting four approaches that are suitable for the model comparison task. We apply two of these methods to a learning analytics dataset from a MOOC conducted at the University of Michigan, providing open-source code in R for reproducing this analysis on other datasets. We offer several practical considerations for future implementations, as well as suggestions for future research in both the learning analytics community and the broader field of machine learning.